Cloudflare, an internet infrastructure and security company, has upgraded its logging pipeline by migrating from syslog-ng to OpenTelemetry Collector. This change affects one of Cloudflare's largest data pipelines, processing millions of log events per second.
Key Points
- Motivations: Language compatibility, easier integration, enhanced metrics, and unified telemetry infrastructure.
- Custom Components: Custom exporter, modified file exporter, processors, and rate-limiters.
- Migration and Future Plans: Careful rollout, monitoring, and plans for more sophisticated log sampling techniques and open-source contributions.
- Other Adopters: Google, Splunk, Shopify, and GitHub are also adopting OpenTelemetry for various use cases.